drivers: bound MSI-X vector use so many-queue guests still boot - #1465
Open
gburd wants to merge 1 commit into
Open
drivers: bound MSI-X vector use so many-queue guests still boot#1465gburd wants to merge 1 commit into
gburd wants to merge 1 commit into
Conversation
The x86-64 IDT exposes only 224 usable interrupt vectors (32..255), and
every device shares that pool. A guest with many vCPUs and several
multiqueue virtio devices can request one MSI-X vector per queue and
exhaust the pool during boot. Two things then went wrong:
- register_interrupt_handler() called abort() when no IDT vector was
free, crashing the boot instead of letting the device degrade.
- request_vectors() pushed a msix_vector whose allocation had failed,
and ~msix_vector() then unregistered vector 0.
Fix both and add a soft cap so devices stay within budget in the first
place:
- register_interrupt_handler() returns 0 (an impossible real vector,
since 0..31 are CPU exceptions) as an "exhausted" sentinel instead
of aborting. msix_vector's destructor skips the unregister when its
vector is 0, and request_vectors() stops at the first failed
allocation and returns the short vector it did get; the existing
easy_register() already treats a short result as failure, so a
driver falls back to fewer queues rather than crashing.
- Add virtio_driver::reserve_msix_vectors(): a process-wide soft
budget that a multiqueue driver consults to cap how many queues it
arms with a distinct interrupt, leaving headroom for other devices
and always granting at least one. virtio-blk uses it to cap its
active queue count; extra probed virtqueues are simply left unused.
Signed-off-by: Greg Burd <greg@burd.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The x86-64 IDT exposes only 224 usable interrupt vectors (32..255), shared by every device. A guest with many vCPUs and several multiqueue virtio devices can request one MSI-X vector per queue and exhaust the pool during boot. When that happened two things went wrong:
register_interrupt_handler()calledabort()when no IDT vector was free, crashing the boot instead of letting the device degrade.request_vectors()pushed amsix_vectorwhose allocation had failed, and~msix_vector()then unregistered vector 0.This bounds vector use so the guest boots either way:
register_interrupt_handler()returns 0 (an impossible real vector, since 0..31 are CPU exceptions) as an "exhausted" sentinel instead of aborting.msix_vector's destructor skips the unregister when its vector is 0, andrequest_vectors()stops at the first failed allocation and returns the short vector it did get. The existingeasy_register()already treats a short result as failure, so a driver falls back to fewer queues rather than crashing.virtio_driver::reserve_msix_vectors(), a process-wide soft budget a multiqueue driver consults to cap how many queues it arms with a distinct interrupt, leaving headroom for other devices and always granting at least one. virtio-blk uses it to cap its active queue count; extra probed virtqueues are simply left unused.Verification
reserve_msix_vectorsresolves within the driver files (defined in virtio.o, referenced in virtio-blk.o); a partial link of the changed objects resolves with no dangling references.